A nomenclature for linear discriminant algorithms

Statistical machine learning focuses on non-parametric models that learn the minimum necessary to perform the targeted discrimination. For instance, in a binary classification problem, SVMs will only use the examples located within a boundary margin between the two classes to compute the optimal separation. However, one can consider other discrimination settings that use the entirety of the data. We first review four different discrimination settings, supported by the example in Figure 1. The task is to separate the ’+’ class from the ’-’ class. Training examples for each classes live in separate horizontal lines. Discrimination always involves positive examples whose scores one wants to push up, and negative examples one wants to push down. This can be expressed in a geometric framework, when one maximizes a separation margin, or in a probabilistic framework, where all scores are normalized to sum to 1, and just pushing up the positive ones necessarily pushes down the negative ones.